Multimodal Cognitive Architecture: Making Perception More Central to Intelligent Behavior

I propose that the notion of cognitive state be broadened from the current predicate-symbolic, Language-of-Thought framework to a multi-modal one, where perception and kinesthetic modalities participate in thinking. In contrast to the roles assigned to perception and motor activities as modules external to central cognition in the currently dominant theories in AI and Cognitive Science, in the proposed approach, central cognition incorporates parts of the perceptual machinery. I motivate and describe the proposal schematically, and describe the implementation of a bi-modal version in which a diagrammatic representation component is added to the cognitive state. The proposal explains our rich multimodal internal experience, and can be a key step in the realization of embodied agents. The proposed multimodal cognitive state can significantly enhance the agent’s problem solving. Cognition, Architecture, Embodiment and Multimodality of Thought Generality and flexibility are hallmarks of intelligence, and this has led to a search for cognitive architectures, exemplified by Soar (Newell, 1990) and ACT-R (Anderson, 1996). Different task-specific cognitive systems may be programmed or modeled by encoding domainand task-specific knowledge in the architecture. They typically posit a working memory (WM), a long term memory (LTM), mechanisms to retrieve from LTM and place in WM information relevant to the task, mechanisms that help the agent set up and explore a problem space, and mechanisms that enable the agent to learn from experience. Proposals for the specific mechanisms along with the representational formalisms on which they work constitute the architecture designer’s theory of cognition. Because of their origin in a certain idealization of human cognition, it is not surprising that Soar and ACT-R are useful both to build AI agents as well to build cognitive models. Copyright © 2006, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. An important aspect of their representational commitment is that the cognitive state, roughly characterized as the content of the WM, is symbolic, or to use a more precise term, predicate-symbolic. That is, the knowledge in LTM as well as representations in WM are compositions of symbol strings where the symbols stand for individuals, relations between individuals, or various ways of composing relational predicates, in the domain of interest. For example, in a blocks world, a state representation might be ON(A,B) & Left(B,C). The commitment to symbolic cognitive state representation extends to almost all of AI (knowledge representation) and Cognitive Science (the Language of Thought hypothesis), i.e., is not restricted to the proposals for a general ExternalWorld/ Representation Perception